Network security and analysis typically uses a variety of discrete network analysis applications to capture, track, and analyze network packet data to protect networks from malicious attacks. Typical network capture applications, which can also be used in applications outside of network security, have limited resources, resulting in a decreased capacity to work in tandem and subsequently provide network administrators the ability to perform multiple analyses in real-time on the same packet flow.
For instance, a packet capture application typically has only enough resources to capture the packets, write all packets to a disk, and apply a light index to allow retrieval of the packets designated by their 5-tuple (a set of five different values comprising a TCP/IP connection, including destination and source IP addresses and port numbers, as well as the particular protocol (for example, TCP or UDP) used by the transmission), which is dictated by a third-party application. Similarly, a flow capture application has only enough resources to inspect the packets and store a flow record of the 5-tuple and any number of additional flow attributes. These resource constraints prevent multiple disparate applications from working on packet flows in a collective manner.
Capture applications are required to inspect packets in real-time for security purposes. In many cases, disparate capture applications inspect the same packet flow, but due to slight differences in packet timing, packet collections cannot be assimilated. Packet timing differs because the capture arrival rates are different from the time the local hardware or software applies a timestamp. Because the timing is different, it is impossible to have 100% accuracy in determining consistent flow identification. A packet flow cannot be uniquely identified by its 5-tuple and timing when packet traffic density may be in the hundreds of thousands of flows per second.